Introduction to the world of containers

Emilio Lario

Introduction to containers

Packages code and all its dependencies.
Runs anywhere with a compatible kernel.

Exercice 1

Go to the VPN and log in to the Linux course machine: vpn
Find a container named “lolcow” from “godlovedc” on Docker Hub:
hub.docker.com
Go back to the Linux terminal and type: singularity run docker://container/name

Solution

  singularity run docker://godlovedc/lolcow

Virtualization and containerization

	Virtual Machines	Container
Guest OS	Each VM runs on virtual hardware and the kernel is loaded into its own memory region.	All containers share the same kernel.
Communication	Through Ethernet Devices.	Standard IPC mechanisms like Signals, pipes, sockets, etc.
Performance	Small overhead as the Machine instructions need to be translated from Guest to Host OS.	Near native performance as compared to the underlying Host OS.
Startup time	Takes up to a few minutes to boot up.	Containers can be booted up in a few seconds.
Isolation	Sharing libraries, files, etc. between guests and between guests and host not natively possible.	Subdirectories can be transparently mounted and can be shared.
Storage	VMs usually require more storage as the whole OS kernel and associated programs have to be installed and run.	Containers consume lower amount of storage since the Host OS is shared.

How containers work?

Namespaces for access control
cgroups for resources management

For a detailed explanation, see Iván Moreno’s article on Medium

Not only Docker

Docker vs Singularity

Sudo in docker

Singularity runs with the same user that starts the container while the Docker daemon runs as root.

Host files access

Default Behavior: Docker isolates the container from the host machine by default. Singularity mounts the /home directory of the user running the container.

Side effects of mounting the home directory

All user customization files will be in the container when it runs. This can break the isolation/reproducibility concept, such as having user-defined R libraries.

It is recommended to run Singularity with –no-home or even with –containall (-C) and bind only the required directories.

Pulling container images

You can create your own container image from ‘scratch,’ but normally you will start with an image that someone else has already created. To do so, you can either run the image straight from the external repository or download it locally and run it later.

Note

Go back to the Linux terminal and type: singularity pull local_image_name.sif docker://container/name to download the image.
Run the local image with: singularity run local_image_name.sif

Solution

  singularity pull lolcow.sif docker://godlovedc/lolcow
  singularity run lolcow.sif

Interactive shell

When we run the container, it loads the image and executes the predefined command, if one is defined. Instead of running the predefined command, we can also open an interactive shell in the container.

Exercice 3

To open an interactive shell in a Singularity container, type: singularity shell local_image_name.sif
(To exit the container, type exit )
Check the user outside and inside the container with: whoami
Check the contents of the home directory outside and inside the container:
Now run the interactive shell with the -C option and verify that the home directory is empty.

Solution

  singularity shell lolcow.sif
  whoami
  exit
  whoami
  ls
  singularity shell lolcow.sif
  ls
  exit
  singularity shell -C lolcow.sif
  ls

Run in a container

We can tell the container to run and execute a command with:

singularity exec local_image_name.sif command

Exercice 4

Check the operating system outside the container with:

head /etc/os-release

Check the operating system inside the container with:

singularity exec local_image_name.sif command

Solution

  head /etc/os-release
  singularity exec lolcow.sif head /etc/os-release

Files access

Persistent changes

All changes made inside a container are lost when the container is stopped. If we need the changes to be persistent, we must use a folder from the host machine.

If we need to access files outside the container, we can “BIND” folders from the host machine to the container:

singularity shell -B /host/path:/path/in/container local_image_name.sif

Exercice 5

Create a folder named “dummy” on the host machine.
Open a terminal inside the container with the -C option (--containall).
Check that there is no “dummy” folder.
Exit the container and enter again, but this time use the bind (-B) option to mount the directory you have created to “/dummy” inside the container.
Create a text file inside the “/dummy” directory inside the container with touch test.txt
Exit the container and check that the file you created inside the container is in the “dummy” directory of the host machine:

Solution

  mkdir dummy
  singularity shell -C lolcow.sif
  ls
  cd /dummy
  exit
  singularity shell -C -B dummy:/dummy lolcow.sif
  cd /dummy
  touch test.txt
  ls
  exit
  cd dummy
  ls

Build a container

scratch
Alpine
Ubuntu
DockerHUb https://hub.docker.com/ –> Rocker, Bioconductor, conda, …

Singularity & Apptainer

From Signulary to Apptainer
“–fakeroot” or remote build

Container recipe

Bootstrap: localimage
From: lolcow.sif

%post
  apt-get -y update
  apt-get -y install sl=3.03-17build1
  apt-get -y install curl

%runscript
  sl -F

Exercice 6

Create a file named steam.def with the above code.
Build the modified image with the command: apptainer build image_name.sif recipe.def
Run the newly created container.

Solution

  vim steam.def
  apptainer build steam.sif steam.def
  singularity run steam.sif

Importance of version reference

Different versions of a dependency may not be 100% compatible.

Containers in the IJC’s HPC

- Account for the resources needed by all the dependencies of your script.

- Do not run a container on the master node!

We are not sudo in the cluster.
Every dependency must be installed by the system administrator.
Run containers with the Slurm scheduler.

Exercice 7

Create a bash script file with the following content:

#!/bin/bash 
curl -s "wttr.in/$1?m1"

Create a Slurm script to execute the script from step 1 inside the container you made in the previous exercise. Copy the following sbatch header and add the command to run the script inside the container. The container will run the command that you type after the image name; therefore, you need to use bash script.sh to actually run the script.

#!/bin/bash 
#SBATCH --job-name=run_lolcow

Use sbatch to run the script and check the output. You can use squeue to see the progress. You will find the log file in the current folder when the job finishes.

Solution

  vim wtt.sh
  
  #!/bin/bash 
  curl -s "wttr.in/$1?m1"
  
  vim wtt.slm
  
  #!/bin/bash 
  #SBATCH --job-name=run_wtt
  singularity exec steam.sif bash wtt.sh
  
  sbatch wtt.slm